Adaptive population optimization for amplifying the intelligence of crowds and swarms

ABSTRACT

System and method amplifying the accuracy of forecasts generated by software systems that harness the collective intelligence of human populations by curating optimized sub-populations through an intelligent selection process. Participants predict event outcomes and/or provide evaluations of their confidence in their predictions. The system determines an outlier score for each participant based on the participant&#39;s responses and the relation of the participant&#39;s responses to the predictions of the population as a whole. Participants can then be selected from the population based on the participant outlier scores.

This application claims the benefit of U.S. Provisional Application No.62/544,861 entitled ADAPTIVE OUTLIER ANALYSIS FOR AMPLYFYING THEINTELLIGENCE OF CROWDS AND SWARMS, filed Aug. 13, 2017, which isincorporated in its entirety herein by reference.

This application claims the benefit of U.S. Provisional Application No.62/552,968 entitled SYSTEM AND METHOD FOR OPTIMIZING THE POPULATION USEDBY CROWDS AND SWARMS FOR AMPLIFIED EMERGENT INTELLIGENCE, filed Aug. 31,2017, which is incorporated in its entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/668,970 entitled METHODS AND SYSTEMS FOR REAL-TIME CLOSED-LOOPCOLLABORATIVE INTELLIGENCE, filed Mar. 25, 2015, now U.S. Pat. No.9,959,028, which in turn claims the benefit of U.S. ProvisionalApplication 61/970,885 entitled METHOD AND SYSTEM FOR ENABLING AGROUPWISE COLLABORATIVE CONSCIOUSNESS, filed Mar. 26, 2014, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/708,038 entitled MULTI-GROUP METHODS AND SYSTEMS FOR REAL-TIMEMULTI-TIER COLLABORATIVE INTELLIGENCE, filed May 8, 2015, which in turnclaims the benefit of U.S. Provisional Application 61/991,505 entitledMETHODS AND SYSTEM FOR MULTI-TIER COLLABORATIVE INTELLIGENCE, filed May10, 2014, both of which are incorporated in their entirety herein byreference.

This application is a continuation-in-part of U.S. application Ser. No.14/738,768 entitled INTUITIVE INTERFACES FOR REAL-TIME COLLABORATIVEINTELLIGENCE, filed Jun. 12, 2015, now U.S. Pat. No. 9,940,006, which inturn claims the benefit of U.S. Provisional Application 62/012,403entitled INTUITIVE INTERFACE FOR REAL-TIME COLLABORATIVE CONTROL, filedJun. 15, 2014, both of which are incorporated in their entirety hereinby reference.

This application is a continuation-in-part of U.S. application Ser. No.14/859,035 entitled SYSTEMS AND METHODS FOR ASSESSMENT AND OPTIMIZATIONOF REAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filed Sep. 18, 2015which in turn claims the benefit of U.S. Provisional Application No.62/066,718 entitled SYSTEM AND METHOD FOR MODERATING AND OPTIMIZINGREAL-TIME SWARM INTELLIGENCES, filed Oct. 21, 2014, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/920,819 entitled SUGGESTION AND BACKGROUND MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Oct. 22, 2015 which in turnclaims the benefit of U.S. Provisional Application No. 62/067,505entitled SYSTEM AND METHODS FOR MODERATING REAL-TIME COLLABORATIVEDECISIONS OVER A DISTRIBUTED NETWORKS, filed Oct. 23, 2014, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.14/925,837 entitled MULTI-PHASE MULTI-GROUP SELECTION METHODS FORREAL-TIME COLLABORATIVE INTELLIGENCE SYSTEMS, filed Oct. 28, 2015 whichin turn claims the benefit of U.S. Provisional Application No.62/069,360 entitled SYSTEMS AND METHODS FOR ENABLING AND MODERATING AMASSIVELY-PARALLEL REAL-TIME SYNCHRONOUS COLLABORATIVESUPER-INTELLIGENCE, filed Oct. 28, 2014, both of which are incorporatedin their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/017,424 entitled ITERATIVE SUGGESTION MODES FOR REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS, filed Feb. 5, 2016 which in turnclaims the benefit of U.S. Provisional Application No. 62/113,393entitled SYSTEMS AND METHODS FOR ENABLING SYNCHRONOUS COLLABORATIVECREATIVITY AND DECISION MAKING, filed Feb. 7, 2015, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/047,522 entitled SYSTEMS AND METHODS FOR COLLABORATIVE SYNCHRONOUSIMAGE SELECTION, filed Feb. 18, 2016 which in turn claims the benefit ofU.S. Provisional Application No. 62/117,808 entitled SYSTEM AND METHODSFOR COLLABORATIVE SYNCHRONOUS IMAGE SELECTION, filed Feb. 18, 2015, bothof which are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/052,876 entitled DYNAMIC SYSTEMS FOR OPTIMIZATION OF REAL-TIMECOLLABORATIVE INTELLIGENCE, filed Feb. 25, 2016 which in turn claims thebenefit of U.S. Provisional Application No. 62/120,618 entitledAPPLICATION OF DYNAMIC RESTORING FORCES TO OPTIMIZE GROUP INTELLIGENCEIN REAL-TIME SOCIAL SWARMS, filed Feb. 25, 2015, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/086,034 entitled SYSTEM AND METHOD FOR MODERATING REAL-TIMECLOSED-LOOP COLLABORATIVE DECISIONS ON MOBILE DEVICES, filed Mar. 30,2016 which in turn claims the benefit of U.S. Provisional ApplicationNo. 62/140,032 entitled SYSTEM AND METHOD FOR MODERATING A REAL-TIMECLOSED-LOOP COLLABORATIVE APPROVAL FROM A GROUP OF MOBILE USERS filedMar. 30, 2015, both of which are incorporated in their entirety hereinby reference.

This application is a continuation-in-part of U.S. application Ser. No.15/199,990 entitled METHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY INA REAL-TIME COLLABORATIVE INTELLIGENCE, filed Jul. 1, 2016, which inturn claims the benefit of U.S. Provisional Application No. 62/187,470entitled METHODS AND SYSTEMS FOR ENABLING A CREDIT ECONOMY IN AREAL-TIME SYNCHRONOUS COLLABORATIVE SYSTEM filed Jul. 1, 2015, both ofwhich are incorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/241,340 entitled METHODS FOR ANALYZING DECISIONS MADE BY REAL-TIMEINTELLIGENCE SYSTEMS, filed Aug. 19, 2016, which in turn claims thebenefit of U.S. Provisional Application No. 62/207,234 entitled METHODSFOR ANALYZING THE DECISIONS MADE BY REAL-TIME COLLECTIVE INTELLIGENCESYSTEMS filed Aug. 19, 2015, both of which are incorporated in theirentirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/640,145 entitled METHODS AND SYSTEMS FOR MODIFYING USER INFLUENCEDURING A COLLABORATIVE SESSION OF REAL-TIME COLLABORATIVE INTELLIGENCESYSTEM, filed Jun. 30, 2017, which in turn claims the benefit of U.S.Provisional Application No. 62/358,026 entitled METHODS AND SYSTEMS FORAMPLIFYING THE INTELLIGENCE OF A HUMAN-BASED ARTIFICIAL SWARMINTELLIGENCE filed Jul. 3, 2016, both of which are incorporated in theirentirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/815,579 entitled SYSTEMS AND METHODS FOR HYBRID SWARM INTELLIGENCE,filed Nov. 16, 2017, which in turn claims the benefit of U.S.Provisional Application No. 62/423,402 entitled SYSTEM AND METHOD FORHYBRID SWARM INTELLIGENCE filed Nov. 17, 2016, both of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/898,468 entitled ADAPTIVE CONFIDENCE CALIBRATION FOR REAL-TIME SWARMINTELLIGENCE SYSTEMS, filed Feb. 17, 2018, which in turn claims thebenefit of U.S. Provisional Application No. 62/460,861 entitledARTIFICIAL SWARM INTELLIGENCE WITH ADAPTIVE CONFIDENCE CALIBRATION,filed Feb. 19, 2017 and also claims the benefit of U.S. ProvisionalApplication No. 62/473,442 entitled ARTIFICIAL SWARM INTELLIGENCE WITHADAPTIVE CONFIDENCE CALIBRATION, filed Mar. 19, 2017, all of which areincorporated in their entirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/904,239 entitled METHODS AND SYSTEMS FOR COLLABORATIVE CONTROL OF AREMOTE VEHICLE, filed Feb. 23, 2018, which in turn claims the benefit ofU.S. Provisional Application No. 62/463,657 entitled METHODS AND SYSTEMSFOR COLLABORATIVE CONTROL OF A ROBOTIC MOBILE FIRST-PERSON STREAMINGCAMERA SOURCE, filed Feb. 26, 2017 and also claims the benefit of U.S.Provisional Application No. 62/473,429 entitled METHODS AND SYSTEMS FORCOLLABORATIVE CONTROL OF A ROBOTIC MOBILE FIRST-PERSON STREAMING CAMERASOURCE, filed Mar. 19, 2017, all of which are incorporated in theirentirety herein by reference.

This application is a continuation-in-part of U.S. application Ser. No.15/922,453 entitled PARALLELIZED SUB-FACTOR AGGREGATION IN REAL-TIMESWARM-BASED COLLECTIVE INTELLIGENCE SYSTEMS, filed Mar. 15, 2018, whichin turn claims the benefit of U.S. Provisional Application No.62/473,424 entitled PARALLELIZED SUB-FACTOR AGGREGATION IN A REAL-TIMECOLLABORATIVE INTELLIGENCE SYSTEMS filed Mar. 19, 2017, both of whichare incorporated in their entirety herein by reference.

This application is a continuation-in-part of International ApplicationNo. PCT/US15/22594, filed Mar. 25, 2015.

This application is a continuation-in-part of International ApplicationNo. PCT/US15/35694, filed Jun. 12, 2015.

This application is a continuation-in-part of International ApplicationNo. PCT/US15/56394, filed Oct. 20, 2015.

This application is a continuation-in-part of International ApplicationNo. PCT/US16/40600, filed Jul. 1, 2016.

This application is a continuation-in-part of International ApplicationNo. PCT/US17/40480, filed Jun. 30, 2017.

This application is a continuation-in-part of International ApplicationNo. PCT/US17/62095, filed Nov. 16, 2017.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates generally to systems and methods forreal-time swarm-based collective intelligence, and more specifically tosystems and methods for selection of users for real-time closed-loopdynamic collaborative control systems.

2. Discussion of the Related Art

The aggregation of insights collected from groups of people has beenshown to amplify intelligence in certain situations. Sometimes calledthe Wisdom of Crowds, these methods generally use statistical averagingacross a static set of data collected from a population. ArtificialSwarm Intelligence is an alternate approach modeled after naturalswarms. It enables populations to aggregate their insights as real-timesystems, with feedback loops, that converge on optimal solutions. Forboth crowd-based and swarm-based methods, it's generally the case thepopulation of participants include a wide range of knowledge levelsand/or skill levels such that some participants provide more insightfulinput than others with respect to the problem at hand. Historicalperformance data for participants is one way that the members of apopulation can distinguished as more or less insightful. But what ifthere is little or no historical information regarding theinsightfulness of participants?

What is needed is inventive systems and methods for enabling theprobabilistic identification of insightful performer versusun-insightful performers within a population based only in their inputdata, without any historical data regarding the accuracy of theirinsights on similar prior tasks, and for amplifying the intelligence ofcrowds (i.e. statistical averages of a population) and/or swarms (i.e.real-time systems created across a population) to enable more accuratepredictions, forecasts, estimations, assessments, and other insights.

SUMMARY OF THE INVENTION

Several embodiments of the invention advantageously address the needsabove as well as other needs by providing a system for curating anoptimized population of human forecasting participants from a baselinepopulation of human forecasting participants based on an algorithmicanalysis of prediction data collected from each participant, theanalysis identifying the likelihood that each participant will be ahigh-performer in a prediction task involving one or more future events,the system comprising: a processing device including a processor andconfigured for network communication; a plurality of applicationinstances wherein each application instance is configured to query aparticipant, receive input from the queried participant about theprediction task, and be in network communication with the processingdevice regarding the prediction task, wherein the system is configuredto perform the steps of: query each member of the baseline population ofparticipants about the prediction task comprised of predicting a set ofevents, wherein each event has a set of possible outcomes including atleast two possible outcomes; collect a set of predictions from eachparticipant, each participant interacting with one application instance,wherein each set of predictions includes a predicted outcome for eachevent of the set of events; for each event in the set of events, computeone or more support values wherein each support value for each eventrepresents the percentage of participants in the baseline populationthat predicted a particular outcome within the set of possible outcomes;for each participant, compute an outlier score for each event, whereinthe outlier score is computed by algorithmically comparing theparticipant's predicted outcome for that event to the support value forthat outcome of that event, wherein the outlier score indicates how wellthat participant's prediction aligns with the predictions given by thebaseline population; for each participant, determine an outlier indexbased on the plurality of outlier scores computed for that participantfor the set of events, the outlier index indicating how well the set ofpredictions provided by that participant aligned with the sets ofpredictions given by the baseline population; curating an optimizedpopulation from the baseline population based at least in part upon aplurality of the outlier indexes, the curation process including atleast one selected from the group of (a) culling a plurality ofparticipants from the baseline population in response to the outlierindex of each culled participant indicating low alignment compared toother participants, and (b) generating a weighting value for a set ofparticipants in the baseline population, the generated weighting valuesbeing lower for participants with an outlier index indicating lowalignment as compared to weighting values for participants with anoutlier index indicating high alignment; and using curated populationinformation to generate at least one crowd-based or swarm-basedprediction for a future event having at least two outcomes.

In another embodiment, the invention can be characterized as A methodfor curating an optimized population of human forecasting participantsfrom a baseline population of human forecasting participants based on analgorithmic analysis of prediction data collected from each participant,the analysis identifying the likelihood that each participant will be ahigh-performer in a prediction task involving one or more future events,comprising the steps of: querying, by a processing device including aprocessor and configured for networked communication, each member of thebaseline population of participants about the prediction task comprisedof predicting a set of events, wherein each event has a set of possibleoutcomes including at least two possible outcomes; collecting, by aplurality of application instances, wherein each application instancereceives input from one participant and is in networked communicationwith the processing device, a set of predictions from each participant,each participant interacting with one application instance, wherein eachset of predictions includes a predicted outcome for each event of theset of events; computing, by the processor for each event in the set ofevents, one or more support values wherein each support value for eachevent represents the percentage of participants in the baselinepopulation that predicted a particular outcome within the set ofpossible outcomes; computing, by the processor for each participant, anoutlier score for each event, wherein the outlier score is computed byalgorithmically comparing the participant's predicted outcome for thatevent to the support value for that outcome of that event, wherein theoutlier score indicates how well that participant's prediction alignswith the predictions given by the baseline population; and determining,by the processor for each participant, an outlier index based on theplurality of outlier scores computed for that participant for the set ofevents, the outlier index indicating how well the set of predictionsprovided by that participant aligned with the sets of predictions givenby the baseline population; curating, by the processor, an optimizedpopulation from the baseline population based at least in part upon aplurality of the outlier indexes, the curation process including atleast one selected from the group of (a) culling a plurality ofparticipants from the baseline population in response to the outlierindex of each culled participant indicating low alignment compared toother participants, and (b) generating a weighting value for a set ofparticipants in the baseline population, the generated weighting valuesbeing lower for participants with an outlier index indicating lowalignment as compared to weighting values for participants with anoutlier index indicating high alignment; and using, by the processor, ofcurated population information to generate at least one crowd-based orswarm-based prediction for a future event having at least two outcomes.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of severalembodiments of the present invention will be more apparent from thefollowing more particular description thereof, presented in conjunctionwith the following drawings.

FIG. 1 is a schematic diagram of an exemplary computing deviceconfigured for use in a collaboration system.

FIG. 2 is a schematic diagram of an exemplary real-time collaborationsystem.

FIG. 3 is a flowchart of an exemplary group collaboration process usingthe collaboration system.

FIG. 4 is a flowchart for a first inventive method for performingAdaptive Outlier Analysis in accordance with one embodiment of thepresent invention.

FIG. 5 is a flowchart for a second inventive method for performingAdaptive Outlier Analysis in accordance with another embodiment of thepresent invention.

FIG. 6 is an exemplary set of question configured for use in embodimentsof the Adaptive Outlier Analysis of the present invention.

FIG. 7 is a flowchart of for an exemplary machine learning trainingphase process in accordance with one embodiment of the presentinvention.

FIG. 8 is a flowchart of for an exemplary machine learning trainingphase process using characterization value in accordance with anotherembodiment of the present invention.

FIGS. 9 and 10 are exemplary probability function charts in oneembodiment of the present invention.

Corresponding reference characters indicate corresponding componentsthroughout the several views of the drawings. Skilled artisans willappreciate that elements in the figures are illustrated for simplicityand clarity and have not necessarily been drawn to scale. For example,the dimensions of some of the elements in the figures may be exaggeratedrelative to other elements to help to improve understanding of variousembodiments of the present invention. Also, common but well-understoodelements that are useful or necessary in a commercially feasibleembodiment are often not depicted in order to facilitate a lessobstructed view of these various embodiments of the present invention.

DETAILED DESCRIPTION

The following description is not to be taken in a limiting sense, but ismade merely for the purpose of describing the general principles ofexemplary embodiments. Reference throughout this specification to “oneembodiment,” “an embodiment,” or similar language means that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment,”“in an embodiment,” and similar language throughout this specificationmay, but do not necessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention can bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

Real-time occurrences as referenced herein are those that aresubstantially current within the context of human perception andreaction.

As referred to in this specification, “media items” refers to video,audio, streaming and any combination thereof. In addition, the audiosubsystem is envisioned to optionally include features such as graphicequalization, volume, balance, fading, base and treble controls,surround sound emulation, and noise reduction. One skilled in therelevant art will appreciate that the above cited list of file formatsis not intended to be all inclusive.

Historical research demonstrates that the insights generated by groupscan be more accurate than the insights generated by individuals in manysituations. A classic example is estimating the number of beans in ajar. Many researchers have shown that taking the statistical average ofestimates made by many individuals will yield an answer that is moreaccurate than the typical member of the population queried. When theindividuals provide their input as isolated data points, to beaggregated statistically, the process is often referred to as“crowdsourcing”. When the individuals form a real-time system andprovide their input together, with feedback loops enabling the group toconverge on a solution in synchrony, the process is often referred to as“swarming”. While very different entities, crowds and swarms both shareone characteristic—a more insightful population will produce moreaccurate insights. Thus there is a significant need for bothcrowdsourcing and swarming to identify a sub-population of highlyinsightful performers from a larger population of general participants.

To put this in context—groups can make predictions about future eventssuch as sporting matches by forming crowds or swarms. Whencrowdsourcing, the process will take the statistical average ofpredictions made by a population of individuals in isolation. The grouppredictions are often mildly more accurate than the individualpredictions. Swarming methods don't use statistical averages of data,but instead form real-time systems that enable the population ofparticipants to work together in real time, with feedback loops thatenable them to converge together on optimized predictions. Such systemshave been shown to significantly outperform crowds. The one thing thatcrowds and swarms have in common is that a population of individualswith more insight about a topic generally produces better predictions ofoutcomes. The present invention is aimed at crafting a more insightfulpopulation by identifying lower-insight participants without havinghistorical information about how those participants perform in priorprediction tasks. By identifying lower performers (and/or by rankinglower vs higher insight performers) the present invention aims to usethis analysis to increase the accuracy of both crowd-based andswarm-based predictions, forecasts, and assessments.

Collaborative Intelligence Systems and Methods

Systems and methods for collaborative swarm intelligence are disclosedin the related applications.

As described in related U.S. Pat. No. 9,959,028 for METHODS AND SYSTEMSFOR REAL-TIME CLOSED-LOOP COLLABORATIVE INTELLIGENCE, by the presentinventor, and incorporated by reference, a swarm-based system andmethods have been developed that enable groups of users tocollaboratively control the motion of a graphical pointer through aunique real-time closed-loop control paradigm. In some embodiments, thecollaboratively controlled pointer is configured to empower a group ofusers to choose letters, words, numbers, phrases, and/or other choicesin response to a prompt posed simultaneously to the group. This enablesthe formation a group response that's not based on the will of anyindividual user, but rather on the collective will of the group. In thisway, the system disclosed herein enables a group of people to expressinsights as a unified intelligence, thereby making decisions, answeringquestions, rendering forecasts, and making predictions as an artificialswarm intelligence.

As described in U.S. patent application Ser. No. 14/708,038 forMULTI-GROUP METHODS AND SYSTEMS FOR REAL-TIME MULTI-TIER COLLABORATIVEINTELLIGENCE, by the current inventor and incorporated by reference,additional systems and methods have been disclosed that encourage groupsof real-time users who are answering questions as a swarm to producecoherent responses while discouraging incoherent responses. A number ofmethods were disclosed therein, including (a) Coherence Scoring, (b)Coherence Feedback, and (c) Tiered Processing. These and othertechniques greatly enhance the effectiveness of the resultingintelligence.

As described in U.S. patent application Ser. No. 14/859,035 for SYSTEMSAND METHODS FOR ASSESSMENT AND OPTIMIZATION OF REAL-TIME COLLABORATIVEINTELLIGENCE SYSTEMS, by the present inventor and incorporated byreference, a system and methods have been developed for enablingartificial swarms to modify its participant population dynamically overtime, optimizing the performance of the emergent intelligence byaltering its population makeup and/or the altering the relativeinfluence of members of within that population. In some such embodimentsthe members of a swarm can selectively eject one or more low performingmembers of that swarm (from the swarm), using the group-wisecollaborative decision-making techniques herein. As also disclosed,swarms can be configured to dynamically adjust its own makeup, not byejecting members of the swarm but by adjusting the relative weighting ofthe input received from members of the swarm. More specifically, in someembodiments, algorithms are used to increase the impact (weighting) thatsome users have upon the closed-loop motion of the pointer, whiledecreasing the impact (weighting that other users have upon theclosed-loop motion of the pointer. In this way, the swarm intelligenceis adapted over time by the underlying algorithms disclosed herein,strengthening the connections (i.e. input) with respect to the morecollaborative users, and weakening the connections with respect to theless collaborative users.

Referring first to FIG. 1, as previously disclosed in the related patentapplications, a schematic diagram of an exemplary portable computingdevice 100 configured for use in the collaboration system is shown.Shown are a central processor 102, a main memory 104, a timing circuit106, a display interface 108, a display 110, a secondary memorysubsystem 112, a hard disk drive 114, a removable storage drive 116, alogical media storage drive 118, a removable storage unit 120, acommunications interface 122, a user interface 124, a transceiver 126,an auxiliary interface 128, an auxiliary I/O port 130, communicationsinfrastructure 132, an audio subsystem 134, a microphone 136, headphones138, a tilt sensor 140, a central collaboration server 142, and acollaborative intent application 144.

Each of a plurality of portable computing devices 100, each used by oneof a plurality of users (the plurality of users also referred to as agroup), is networked in real-time to the central collaboration server(CCS) 142. In some embodiments, one of the portable computing devices100 could act as the central collaboration server 142. For the purposesof this disclosure, the central collaboration server 142 is its owncomputer system in a remote location, and not the portable computingdevice 100 of one of the users. Hence the collaboration system iscomprised of the centralized central collaboration server 142 and theplurality of portable computing devices 100, each of the portablecomputing devices 100 used by one user.

The portable computing device 100 may be embodied as a handheld unit, apocket housed unit, a body worn unit, or other portable unit that isgenerally maintained on the person of a user. The portable computingdevice 100 may be wearable, such as transmissive display glasses.

The central processor 102 is provided to interpret and execute logicalinstructions stored in the main memory 104. The main memory 104 is theprimary general purpose storage area for instructions and data to beprocessed by the central processor 102. The main memory 104 is used inthe broadest sense and may include RAM, EEPROM and ROM. The timingcircuit 106 is provided to coordinate activities within the portablecomputing device 100. The central processor 102, main memory 104 andtiming circuit 106 are directly coupled to the communicationsinfrastructure 132. The central processor 102 may be configured to run avariety of applications, including for example phone and address bookapplications, media storage and play applications, gaming applications,clock and timing applications, phone and email and text messaging andchat and other communication applications. The central processor 102 isalso configured to run at least one Collaborative Intent Application(CIA) 144. The Collaborative Intent Application 144 may be a standaloneapplication or may be a component of an application that also runs uponother networked processors.

The portable computing device 100 includes the communicationsinfrastructure 132 used to transfer data, memory addresses where dataitems are to be found and control signals among the various componentsand subsystems of the portable computing device 100.

The display interface 108 is provided upon the portable computing device100 to drive the display 110 associated with the portable computingdevice 100. The display interface 108 is electrically coupled to thecommunications infrastructure 132 and provides signals to the display110 for visually outputting both graphics and alphanumeric characters.The display interface 108 may include a dedicated graphics processor andmemory to support the displaying of graphics intensive media. Thedisplay 110 may be of any type (e.g., cathode ray tube, gas plasma) butin most circumstances will usually be a solid state device such asliquid crystal display. The display 110 may include a touch screencapability, allowing manual input as well as graphical display.

Affixed to the display 110, directly or indirectly, is the tilt sensor140 (accelerometer or other effective technology) that detects thephysical orientation of the display 110. The tilt sensor 140 is alsocoupled to the central processor 102 so that input conveyed via the tiltsensor 140 is transferred to the central processor 102. The tilt sensor140 provides input to the Collaborative Intent Application 144, asdescribed later. Other input methods may include eye tracking, voiceinput, and/or manipulandum input.

The secondary memory subsystem 112 is provided which houses retrievablestorage units such as the hard disk drive 114 and the removable storagedrive 116. Optional storage units such as the logical media storagedrive 118 and the removable storage unit 118 may also be included. Theremovable storage drive 116 may be a replaceable hard drive, opticalmedia storage drive or a solid state flash RAM device. The logical mediastorage drive 118 may be a flash RAM device, EEPROM encoded withplayable media, or optical storage media (CD, DVD). The removablestorage unit 120 may be logical, optical or of an electromechanical(hard disk) design.

The communications interface 122 subsystem is provided which allows forstandardized electrical connection of peripheral devices to thecommunications infrastructure 132 including, serial, parallel, USB, andFirewire connectivity. For example, the user interface 124 and thetransceiver 126 are electrically coupled to the communicationsinfrastructure 132 via the communications interface 122. For purposes ofthis disclosure, the term user interface 124 includes the hardware andoperating software by which the user executes procedures on the portablecomputing device 100 and the means by which the portable computingdevice 100 conveys information to the user. In the present invention,the user interface 124 is controlled by the CIA 144 and is configured todisplay information regarding the group collaboration, as well asreceive user input and display group output.

To accommodate non-standardized communications interfaces (i.e.,proprietary), the optional separate auxiliary interface 128 and theauxiliary I/O port 130 are provided to couple proprietary peripheraldevices to the communications infrastructure 132. The transceiver 126facilitates the remote exchange of data and synchronizing signalsbetween the portable computing device 100 and the Central CollaborationServer 142. The transceiver 126 could also be used to enablecommunication among a plurality of portable computing devices 100 usedby other participants. In some embodiments, one of the portablecomputing devices 100 acts as the Central Collaboration Server 142,although the ideal embodiment uses a dedicated server for this purpose.In one embodiment the transceiver 126 is a radio frequency type normallyassociated with computer networks for example, wireless computernetworks based on BlueTooth® or the various IEEE standards802.11.sub.x., where x denotes the various present and evolving wirelesscomputing standards. In some embodiments the portable computing devices100 establish an ad hock network between and among them, as with aBlueTooth® communication technology.

It should be noted that any prevailing wireless communication standardmay be employed to enable the plurality of portable computing devices100 to exchange data and thereby engage in a collaborative consciousnessprocess. For example, digital cellular communications formats compatiblewith for example GSM, 3G, 4G, and evolving cellular communicationsstandards. Both peer-to-peer (PPP) and client-server models areenvisioned for implementation of the invention. In a third alternativeembodiment, the transceiver 126 may include hybrids of computercommunications standards, cellular standards and evolving satelliteradio standards.

The audio subsystem 134 is provided and electrically coupled to thecommunications infrastructure 132. The audio subsystem 134 is configuredfor the playback and recording of digital media, for example, multi ormultimedia encoded in any of the exemplary formats MP3, AVI, WAV, MPG,QT, WMA, AIFF, AU, RAM, RA, MOV, MIDI, etc.

The audio subsystem 134 in one embodiment includes the microphone 136which is used for the detection and capture of vocal utterances fromthat unit's user. In this way the user may issue a suggestion as averbal utterance. The portable computing device 100 may then capture theverbal utterance, digitize the utterance, and convey the utterance toother of said plurality of users by sending it to their respectiveportable computing devices 100 over the intervening network. In thisway, the user may convey a suggestion verbally and have the suggestionconveyed as verbal audio content to other users. It should be noted thatif the users are in close physical proximity the suggestion may beconveyed verbally without the need for conveying it through anelectronic media. The user may simply speak the suggestion to the othermembers of the group who are in close listening range. Those users maythen accept or reject the suggestion using their portable electronicdevices 100 and taking advantage of the tallying, processing, andelectronic decision determination and communication processes disclosedherein. In this way the present invention may act as a supportivesupplement that is seamlessly integrated into a direct face to faceconversation held among a group of users.

For embodiments that do include the microphone 136, it may beincorporated within the casing of the portable computing device 100 ormay be remotely located elsewhere upon a body of the user and isconnected to the portable computing device 100 by a wired or wirelesslink. Sound signals from microphone 136 are generally captured as analogaudio signals and converted to digital form by an analog to digitalconverter or other similar component and/or process. A digital signal isthereby provided to the processor 102 of the portable computing device100, the digital signal representing the audio content captured bymicrophone 136. In some embodiments the microphone 136 is local to theheadphones 138 or other head-worn component of the user. In someembodiments the microphone 136 is interfaced to the portable computingdevice 100 by a Bluetooth® link. In some embodiments the microphone 136comprises a plurality of microphone elements. This can allow users totalk to each other, while engaging in a collaborative experience, makingit more fun and social. Allowing users to talk to each other could alsobe distracting and could be not allowed.

The audio subsystem 134 generally also includes headphones 138 (or othersimilar personalized audio presentation units that display audio contentto the ears of a user). The headphones 138 may be connected by wired orwireless connections. In some embodiments the headphones 138 areinterfaced to the portable computing device 100 by the Bluetooth®communication link.

The portable computing device 100 includes an operating system, thenecessary hardware and software drivers necessary to fully utilize thedevices coupled to the communications infrastructure 132, media playbackand recording applications and at least one Collaborative IntentApplication 144 operatively loaded into main memory 104, which isdesigned to display information to a user, collect input from that user,and communicate in real-time with the Central Collaboration Server 142.Optionally, the portable computing device 100 is envisioned to includeat least one remote authentication application, one or more cryptographyapplications capable of performing symmetric and asymmetriccryptographic functions, and secure messaging software. Optionally, theportable computing device 100 may be disposed in a portable form factorto be carried by a user.

Referring next to FIG. 2, a collaboration system 200 is shown in oneembodiment of the present invention. Shown are the central collaborationserver 142, a plurality of the secondary memory subsystems 112, aplurality of the timing circuits 106, a first portable computing device202, a second portable computing device 204, a third portable computingdevice 206, and a plurality of exchanges of data 208.

The group of users (participants), each using one of the plurality ofportable computing devices 100, each portable computing device 100running the Collaborative Intent Application 144, each device 100 incommunication with the Central Collaboration Server 142, may engage inthe collaborative experience that evokes a collective intelligence (alsoreferred to as Collective Consciousness).

As shown in FIG. 2, the CCS 142 is in communication with the pluralityof portable computing devices 202, 204, 206. Each of these devices 202,204, 206 is running the Collaborative Intent Application (CIA) 144. Inone example, each of the devices 202, 204, 206 is an iPad® running theCIA 144, each iPad® communicating with the CCS 142 which is running aCollaboration Mediation application (CMA) 210. Thus, we have the localCIA 144 on each of the plurality of devices 202, 204, 206, each device202, 204, 206 in real-time communication with the CMA running on the CCS142. While only three portable devices 202, 204, 206 are shown in FIG. 2for clarity, in ideal embodiments, dozens, hundreds, thousands, or evenmillions of devices 100 would be employed in the collaboration system200. Hence the CCS 142 must be in real-time communication with manydevices 100 at once.

The communication between the CCS 142 and each of the devices 202, 204,206 includes the exchanges of data 208. The data has a very significantreal-time function, closing the loop around each user, over theintervening electronic network.

As described above, the present invention allows the group of users,each using their own tablet or phone or other similar portable computingdevice 100, to collaboratively answer questions in real-time with thesupport of the mediating system of the CCS 142 which communicates withthe local CIA 144 running on each device 100. The Collaborative IntentApplication 144 ties each device 100 to the overall collaborative system200. Multiple embodiments of the CIA 144 are disclosed herein. TheCollaborative Intent Application (CIA) 144 may be architected in avariety of ways to enable the plurality of portable computing devices100 to engage in the collaborative processes described herein, with thesupportive use of the Central Collaboration Server 142.

In some embodiments the exchange of data 208 may exist between portablecomputing devices 100.

Referring next to FIG. 3, a flowchart of one embodiment of a groupcollaboration process is shown. Shown are a collaboration opportunitystep 300, a user input step 302, a send user intents to CCS step 304, adetermine group intent step 306, a send group intent to CIA step 308, adisplay intents step 310, a target selection decision point 312, and adisplay target step 314. The process also includes optional steps thatcould be included, for example, for a pointer graphical embodiment: adisplay pointer start position step 316, a display input choices step318, and an update pointer location step 320. In the collaborationopportunity step 300, the CIA 144 receives the group collaborationopportunity from the CCS 142 and displays the opportunity on the display110 of the portable computing device 100 (PCD). The group collaborationopportunity may be a question to be answered, for example, “What filmwill win the Best Picture in the Academy Awards?” or “Who will win theSuper Bowl?” The process then proceeds to the user input step 302. Theuser input step 302 includes the user using the display 110 of thecomputing device 100 to input the user intent. The user intent is aninput interpreted by the user interface 124 as a desired vectordirection conveying an intent of the user. In some embodiments(described in the related applications), the user intent is a desiredvector direction of the graphical pointer 406 of the user interface 124,and the user input includes swiping of the pointer 406 via thetouchscreen interface. The user input step 302 takes place for each userof the group. The process then proceeds to the send user intent to CCSstep 304.

In the send user intent to CCS step 304, the CIA 144 for each PCD 100sends the user intent to the CCS 142. In the next step, the determinegroup intent step 306, the CCS 142 determines a collective group intentbased on the plurality of user intents. The group intent may bedetermined through various methods, as described further below. Theprocess then proceeds to the send group intent to CIA step 308.

In the embodiment including the optional steps display pointer startposition 316 and the display input choices step 318, in the displaypointer start position step 316 the graphical user interface 124 woulddisplay the starting, or neutral, position of a pointer chosen toindicate the graphical representation of the group intent. In thefollowing step, the display input choices step 318, the user interface124 would display a plurality of input choices 412 available to beselected by the group intent by using the pointer. The user intent inthis embodiment is an input interpreted by the user interface 124 asrepresenting that user's desired motion of the collaborative graphicalpointer with respect to the plurality of input choices.

In the send group intent to CIA step 308, the CIA 144 receives the groupintent from the CCS 142. Next, in the display intents step 310, for eachcomputing device 100 the received representation of the group intent isdisplayed, along with a representation of the user intent originallyinput by the user of the computing device 100. The process then proceedsto the target selection decision point 312.

The update pointer location step 320 may be inserted between the displayintents step 310 and the target selection decision point 312. In theupdate pointer location step 320, in the embodiments including thepointer 410 the user interface 124 updates to indicate the currentlocation of the pointer in response to the received group intent.

In the target selection decision point 312, if the group intent receivedcorresponds to selection of the target (in some embodiments, from amongthe input choices), the process proceeds to the display target step 314,and the selected target is displayed on the display 124. If the groupintent has not selected the target, the process returns to the userinput step 302, and the process repeats until the target is determinedby the group intent or until the process is otherwise ended (forexample, by a time limit).

After the target has been chosen by the group intent, the entire processmay repeat, for example, to form a word if each consecutive target is analphabetic character.

Referring again to FIGS. 1, 2 and 3, the collaboration system in oneembodiment as previously disclosed in the related applications employsthe CCS 142 that users connect to via their portable computing device100. In some embodiments, fixed or non-portable computing devices 100can be used as well. In many embodiments, users choose or are assigned ausername when they log into the CCS 142, thus allowing software on theCCS 142 to keep track of individual users and assign each one a scorebased on their prior sessions. This also allows the CCS 142 to employuser scores when computing the average of the group intent of all theusers (in embodiments that use the average).

In general, when the session is in progress, the question is sent fromthe CCS 142 to each of the CIA 144 on the portable computing devices 100of the users. In response to the question, the users convey their ownintent either by manipulating an inner puck of the pointer, as describedin the related applications, or by using a tilt or swipe input or otheruser interface methods. In some embodiments, the user's intent isconveyed as a direction and a magnitude (a vector) that the user wantsthe pointer to move. This is a user intent vector and is conveyed to theCCS 142. In some embodiments, the magnitude of the user intent vector isconstant. The CCS 142 in some embodiments computes the numerical average(either a simple average or a weighted average) of the group intent forthe current time step. Using the numerical average, the CCS 142 updatesfor the current time step the graphical location of the pointer within atarget board displayed on the display 110. This is conveyed as anupdated coordinate location sent from the CCS 142 to each of the CIA 144of participating users on their own devices 100. This updated locationappears to each of the users on their individual devices 100. Thus theysee the moving pointer, ideally heading towards an input choice on thetarget board. The CCS 142 determines if and when the input choice issuccessfully engaged by the pointer and if so, that target is selectedas an answer, or as a part of the answer (a single letter or space orpunctuation mark, for example, that's added to an emerging answer). Thattarget is then added to the emerging answer, which is sent to all thedevices 100 and appears on each display 110.

While FIG. 3 illustrates one embodiment of the collaborative process, asshown in the related applications, many variations of the basic processare contemplated by the inventor.

Adaptive Outlier Analysis

While the systems described above enable human groups to amplify theircollective intelligence and produce optimized predictions, forecasts,and decisions, what is needed are inventive methods that furtheroptimize the intelligence of an Artificial Swarm Intelligencecollaborative system so that it generates predictions, forecasts, anddecisions of even greater accuracy. Specifically, what is needed is amethod of distinguishing members of the population (i.e. humanforecasting participants) that are likely to be high-insight performers(i.e. high performers) on a given prediction task as compared to membersof the population who are likely to be low-insight performers (i.e. lowperformers) on a given prediction task, and doing so without usinghistorical data about their performance on similar tasks. That's becausethere are many situations where historical data does not exist aboutuser performance when providing insights in response to a query. Themethod disclosed herein is called Adaptive Outlier Analysis and itenables both crowd and swarm-based predictions to increase theiraccuracy in many situations.

When harnessing the intelligence of a CROWD (i.e. an open loop group) ora SWARM (i.e. a closed-loop system) it can be beneficial to curate thepool of human participants so as to increase their statisticallikelihood of giving accurate insights in response to a query. Thepresent invention enables the curation of human participants in twoinventive forms—(I) taking an initial pool of baseline forecastingparticipants (i.e. a baseline population) and culling that pool down toa final pool of curated forecasting participants (i.e. an optimizedforecasting population) which are then used for crowd-based orswarm-based intelligence generation, and/or (II) taking a pool ofbaseline forecasting participants and assigning weighting factors tothose participants based on their likelihood of giving accurate insights(i.e. giving a higher weight within a swarm to participants who aredetermined to be more likely to give accurate insights than participantswho are determined to be less likely to give accurate insights). In someinventive embodiments, both culling and weighting are used incombination—giving a curated pool that has eliminated the participantswho are most likely to be low insight performers, and weighting theremaining members of the pool based on their likelihood of beingaccurate insight performers.

To perform the culling and/or weighting process, many methods arepossible. The present invention is focused on an inventive method thatdoes not require any previous historical data about the insight level ofthe participants when performing a similar task (i.e. it does notrequire historical data about the accuracy of the insights provided byparticipants on prior tasks for which the outcome is already known).That said, other inventive methods by the present inventor do usehistorical performance, for example, as disclosed in previouslymentioned U.S. patent application Ser. No. 14/859,035.

Instead, the present invention is focused on a process referred toherein as Adaptive Outlier Analysis which uses only the set of insights(i.e. predictions) given by each participant of a population, withoutknowledge of whether those insights (i.e. predictions) are correct orincorrect because the outcome of the event is still pending. Forexample, when predicting a set of 10 baseball games—a group ofparticipants can provide insights in the form of predictions (i.e. a setof prediction data) for the winner of each those 10 baseball games. Thepresent invention is aimed at identifying those users who are mostlikely to be low-insight performers (i.e. low performers who have lowaccuracy in predicting the 10 games) as compared to those users who aremost likely to be high-insight performers (i.e. high performers who havehigh accuracy in predicting the 10 games). Thus before the games areplayed (i.e. without knowing which predictions are accurate and whichare not accurate), the present invention enables the statisticalidentification of participants who are most likely to be less accurateperformers compared to other participants who are most likely to be moreaccurate performers. Once this identification is made, the group (i.e.the baseline population) can be culled and/or weighted accordingly,thereby creating an optimized population for use in crowd-based and/orswarm-based forecasting processes.

Referring next to FIG. 4, a flowchart for a first inventive method forperforming Adaptive Outlier Analysis is shown. The method begins in step400 wherein a population of users are invited to engage in the AdaptiveOutlier Analysis process, said users being networked users who interactwith online resources. In some embodiments the system used is thecollaboration system 200 of FIG. 2, where the population of users aremembers of the collaboration system 200 (wherein the centralcollaboration server functions as the survey platform andqueries/collects information from the participants). In some embodimentsthe population of users is invited to an online cloud-based platformthrough an invitation provided on Facebook, Twitter, or other similarsocial networks. In some embodiments the population of users is invitedto an online cloud-based platform through a direct email, direct textmessage, or other similar messaging technology. In some embodiments thepopulation of users is invited to an online cloud-based platform througha digital advertisement presented on Facebook®, Twitter®, Reddit®,Google® AdWords®, or other suitable advertising mechanism. In someembodiments the population of users is acquired through a human samplingservice or through a distributed worker service like Amazon MechanicalTurk®. However the population of users is targeted with an invitation,users choosing to participate are routed to an online survey system thatcollects personal data and individual predictions from the population.The online survey system can be a traditional commercial survey platformlike Survey Monkey® or could be a conversational survey platform thatcollects data through text and/or verbal conversations with chat bots oreven live moderators. As the platform used to interact with the userscan take many forms (browser-based, mobile application, desktopapplication, etc.), the instances of programming used to query eachparticipant and communicate with the processor used for analysis isreferred to as an “application instance”.

In the next collect information step 402, regardless of the datacollection method employed, each of the population of participantsresponding to the invitation (also referred to as a baseline populationof participants) is provided by the application instance with a set ofqueries and each participant is expected to provide a set of answers,the answers stored as accessible data. The questions may includepersonal contact information, personal demographic and psychographicinformation, and other personal metadata. For example, each user in apopulation engaged in a task to predict sporting events may be asked fortheir name, age, location, highest level of education, and politicalaffiliation. They may also be asked to identify which sports theyfollow, which teams they are fans of, and how much time they spendfollowing particular sports each week. They may also be asked toself-assess themselves on a subjective rating scale, for example beingasked to rate their level of sports knowledge for a particular sport,rate their predictive skills for a particular sport, and rate theirperceived abilities as compared to the average sports fan.

In the next collect predictions step 404, the survey platform(application instance) requires each member of the population to providea set of predictions for a set of events, wherein each event in the setof events has a set of possible outcomes including at least two possibleoutcomes. The set of predictions, for example, may be a set of 10 sportsgames to be played on a particular day, each of which has three possibleoutcomes—Team A wins, Team B wins, the teams tie. Thus for each of the10 sports games to be played on the particular day, each participant isasked to indicate their prediction (i.e. indicate if Team A wins, Team Bwins, or if it's a tie). In some embodiments, each participant is alsoasked to rate their subjective confidence in each prediction onsubjective rating scale, for example a scale from 0% to 100%. In someembodiments, each participant is asked to predict what percentage ofother participants they expect will predict the same outcome as theypredicted. For example, if a particular participant predicts that Team Bwill win, the participant will also predict that they have a confidenceof 75% and that they expect 40% of the other participants will predictthe same way that they have. They may also indicate that they would bet$25 out of a maximum of $100 on this particular outcome, as a furthermeans of indicating their confidence in the predicted outcome. At theend of step 103, a set of data is collected for each of the participantsin the population of participants, the data collected for eachparticipant including a prediction of an outcome for each of the set ofevents, as well as one or more confidence metrics reflecting theirabsolute and/or relative confidence in the predicted outcome of theparticular event. This set of participant data (both predictions andconfidence metrics) is referred to herein as the Participant PredictionDataset.

In the compute Outcome Support step 406, for each event in the set ofevents, the data in the Participant Prediction Dataset is processed bythe main processor (in some embodiments the CCS 142) to determine thepercentage of participants who predicted each of the possible outcomesfor that particular event in the set of events. For example, it could bedetermined in a population of 200 sports fans who are predicting anEnglish Premier League soccer game between Arsenal and Liverpool that70% (i.e. 140 participants) predicted Arsenal will win, 20% (i.e. 40participants) predicted Liverpool will win, and 10% (i.e. 20participants) predicted the teams would tie. This is referred to as theOutcome Support Percentages for this particular event, with 70% supportfor the prediction that Arsenal will win, 20% support for the predictionthat Liverpool will win, and 10% support for the prediction that the twoteams will tie. Outcome Support Percentages are computed in this stepfor each event in the set of events. For example, Outcome SupportPercentages may be computed for all 10 matches played on a given weekendwithin the English Premier League. In some embodiments, a scaled ornormalized version of the Outcome Support Percentages is computed,referred to as the Outcome Support Indexes. For example, in oneembodiment, the participants who support a particular outcome areweighted based on the level of confidence they have indicated in one ormore confidence metrics associated with their predictions. In suchembodiments, a highly confident participant will count more in theOutcome Support Percentage computation than a less confidentparticipant. While the output of step 406 can be the simple OutcomeSupport Percentage or the more sophisticated Outcome Support Index, theprocess proceeds similarly to step 408 wherein the Outcome Supportvalues (Percentage or Index) are used to determine Outlier Scores foreach participant. Outcome Support values are also referred to herein assupport values.

In step 408, an Outlier Score is computed by the processor for eachparticipant in the population of participants based on (a) thepredictions made by that participant for the set of events and (b) theOutcome Support values computed for each corresponding event within theset of events. The Outlier Score for a given user and a given event iscomputed such that a higher score is assessed when a user predicts anoutcome for that event that has a low percentage (or a low index) in theOutcome Support values. Accordingly, the Outlier Score for a given userand a given event is computed such that a lower score is assessed when auser predicts an outcome for that event that has a high percentage (orhigh index) in the Outcome Support values. For example, if a userpredicts that Arsenal will win the game between Arsenal and Liverpool,and the Outcome Support Percentage for Arsenal winning is 70%, the userpredicted an outcome that has a high degree of support within thepopulation of participants. This corresponds with a low Outlier Scorefor this user for this game. Conversely, if the user predicts that theteams will tie, and the Outcome Support Percentage for the game endingin a tie is 10%, the user predicted an outcome that has a low degree ofsupport within the population of participants. This corresponds with ahigh Outlier Score for this user for this game. In fact, this user wouldbe considered an outlier within the population, for he or she predictedan outcome that had very low support within the overall population.

A variety of algorithmic methods have been employed to compute OutlierScores. In one embodiment, the outlier score is computed as(100%−Outcome_Support)²

Where Outcome_Support is the Outcome Support Percentage that correspondswith the outcome predicted by the given participant.

For example, in the case described above wherein a user predicts Arsenalwinning, and wherein the Outcome Support Percentage for Arsenal is 70%,the Outlier Score is computed as (100%−70%)²=(0.3)²=0.09, or 9%. This isa very low Outlier Score for this event because the user predicted anoutcome that had very high support within the population. In the othercase described above wherein the user predicts a tie and wherein theOutcome Support Percentage for a tie is 10%, the Outlier Score iscomputed as (100%−10%)²=(1−0.1)²=0.81, or 81%. This is a very highOutlier Score for this event because the user predicted an outcome thathad very low support within the population. In this step 308, each userin the population of participants is assigned an Outlier Score for eachprediction made by that user in the set of events. Thus if there were100 participants and 10 events, 1000 outlier scores are generated (tenoutlier scores for each of the 100 participants).

Referring next to step 410, an Outlier Index is computed by theprocessor for each participant in the population of participants, theOutlier Index being based on the participant's Outlier Score for each ofthe events within the set of events. Thus if there were 10 events in theset of events, the Outlier Index for each participant is based on thatparticipant's Outlier Score for each of the 10 events. In this way, theOutlier Index indicates the participant's general trend in OutlierScoring across all the events in a given set of events. The OutlierIndex can be computed in a variety of inventive ways. In someembodiment, the Outlier Index is computed as the average Outlier Scoreassigned to that participant across the set of events. For example, ifthere were 10 events in the set of events, the participant would havebeen assigned 10 outlier scores in the prior step (step 308). In thisstep 310, the participant would be assigned an Outlier Index equal tothe mean of the outlier scores assigned to those ten events. In otherembodiments, the Outlier Index is computed as a weighted average, wherethe outlier scores are weighted based on the confidence metrics that aparticipant associated with each of the predictions in the set ofevents. An outlier score associated with a high confidence metric isweighted strongly in the Outlier Index, while an outlier scoreassociated with a low confidence metric is weighted weakly in theOutlier Index. This has an inventive benefit, for it gives an OutlierIndex that rewards users with a low contribution to their Outlier Indexif they had low confidence in a pick that had a high Outlier Score, asthey knew that they were making an unusual pick. Conversely, it gives anOutlier Index that penalizes users with a high contribution to the indexif they had high confidence in a pick that earned a high Outlier Score,as they likely did not know they were making an unusual pick. Thus, thisinventive method works to reward participants who are self-aware whentheir predictions are likely to be contrarian, and penalizesparticipants are unaware when their predictions are likely to becontrarian. Note—statistically speaking, predictions that have a lowOutcome Support Percentage are generally riskier predictions, and thusshould be associated with lower confidence metrics across a large numberof events.

Referring next to step 412, once an Outlier Index has been generated foreach of the participants in the population of participants, a curatedsub-population is determined by the processor from the full populationof participants. The curated sub-population is generated based at leastin part on the Outlier Index values generated for each of theparticipants in the full population. In some embodiments, other valuesare used as well, for example user meta-data regarding each user'sself-assessment of knowledge in the subject at hand, either absolute orrelative to the average participant in the population. The inventiveprocess of curating a sub-population involves the inventive steps of (a)removing participants from the population based at least in part upontheir Outlier Index being above a certain value, and/or (b) generating aweighting factor for participants within the population, the weightingfactor based at least in part upon that user's Outlier Index as comparedto other user's Outlier Indices. Said weighting factors are then used bysubsequent crowd-based and/or swarm-based methods for generatinginsights from the group. For example, in crowd-based methods thatcompute a statistical average of predictions, the weighting factors areused to weight each user's predictive contribution to that statisticalaverage. Or, for example, in swarm-based methods that enable a real-timecontrol system, the weighting factors are used to weight each user'sinfluence upon the real-time control system.

In some embodiments, a statistical distribution of Outlier Scores isgenerated for the full population of users in the population ofparticipants. In some such embodiments, a threshold level within thatstatistical distribution is computed such that any users with an OutlierScore above the threshold level are not included in the sub-populationthat is used for crowd-based or swarm-based predictions. In someembodiments the statistical distribution is a reasonable approximationof a normal distribution, and the threshold is defined in terms of astandard deviation. In one such embodiment, users who have an outlierindex that is more than 1 standard deviation above the mean, are notincluded in the curated sub-population.

Referring next to FIG. 5, a flowchart for another embodiment of theinventive method for performing Adaptive Outlier Analysis is shown,wherein the event outcomes are binary.

There is a large class of events wherein the predictive outcomes haveonly two conditions, i.e. the event outcomes are binary. For example,baseball games in the US have only two outcomes—Team A wins or Team Bwins. Similarly, some financial predictions can be represented withbinary outcomes as well—price goes up, or price goes down. In suchsituations, a simplified method can be used that reduces thecomputational burden of the process associated with FIG. 4. In thissimplified method, shown in FIG. 5, the Outcome Support Percentages canbe compared against a Super-Majority Threshold percentage, below whichthere's not enough consensus in an outcome to assign any scorecontributions to an outlier index. In one preferred embodiment, theSuper-Majority Threshold percentage is approximately 68%, which has beendetermined experimentally to provide highly effective outcomes.

In FIG. 5, the binary Adaptive Outlier Analysis process starts withsteps 500, 502, and 504, which are similar to the description of events400, 402 and 404 with respect to FIG. 4. In these steps, a Population ofParticipants is engaged for a Set of Events, but in this case, each ofthe set of events has only two possible outcomes. For example—Team Awins or Team B wins. Thus in step 504, a set of predictions is collectedfrom each member of the Population of Participants, each prediction inthe set being an indication of which of the two possible outcomes theparticipant believes is most likely. In addition to collecting outcomepredictions, the method may also include the collection of confidencemetrics for the predicted outcome—for example, a subjective rating ofconfidence on a scale of 0 to 100%. In some embodiments, the user isasked to predict the odds that their chosen outcome will be the correctchosen outcome on a scale of 0 to 100%. In some embodiments, aconfidence metric is used such that the participant is asked to predictwhat percentage of other participants will choose the same predictionthat he or she did on a scale of 0 to 100%. In some embodiments, theparticipant is asked to assign a wager to the prediction, on a range of$0 to $100, as an indication of confidence.

In Determine Majority Outcome Percentage step 506, the inventive methoddetermines which of the two outcomes received the majority ofparticipant predictions, as well as computing the percentage ofparticipants who provided this majority prediction. This value is storedas the Majority Outcome Percentage for that event. For example, if theoutcome of Team A winning received the majority of predictions for Event5 at a level of 78% of participants, then the Majority Outcome of event5 is Team A, and the Majority Outcome Percentage for event 5 is 78%.Accordingly, the minority outcome would be Team B winning.

In step 508, for each event in the set of events, the Majority OutcomePercentage is compared against the Super-Majority Threshold value. Ifthe Majority Outcome Percentage is greater than or equal to the MajorityThreshold value, a Super Majority indicator is assigned to that event.For example, in the hypothetical “event 5” described in step 406 above,because 78% of participants predicted a Team A win, and because 78%exceeds the Super Majority Threshold of 68%, then event 5 is deemed aSuper Majority event, supporting Team A. In practice, the Super MajorityThreshold is chosen to be a high enough value that the majority issignificant, but a low enough value that a plurality of events in theset of events are assigned Super Majority status. Ideally, 40% to 60% ofevents within a set of events earn Super Majority status, although itcan fall outside that range. And if a large number of events are beingpredicted in the set, a higher threshold can be used, reducing thepercentage of supermajorities.

In step 510, Outlier Scores are assigned to each participant of eachSuper Majority event. The scoring is such that if a participantpredicted the Majority Outcome of a Super Majority event, they receivean Outlier Score=0 for that event, but if the participant did notpredict the Majority Outcome of the Super Majority event, they receivean Outlier Score=1 for that event. This method ensures that outlierscores, across a set of events, will accrue for participants whoconsistently predict outcomes other than the Majority Outcome for SuperMajority events. In some embodiments, the Outlier Score assigned forSuper Majority events is scaled based on (a) one or more confidencemetrics provided by the user for the prediction of the event, and/or (b)the Majority Outcome Percentage (for example, the amount by which theSuper Majority Percentage exceeded the Super Majority Threshold). Insuch embodiments, the Outlier Score is lower for a participant whoindicated low confidence in the prediction, and the Outlier score ishigher for a participant who indicated high confidence in theprediction. In one embodiment, the Outlier Score for a user who predictsan outcome that is not the Majority Outcome is(1*Confidence_Percentage)²

where Confidence_Percentage is a percentage based on the confidencemetrics provided by the participant. In this embodiment, the higher theconfidence percentage provided, the higher the Outlier Score.

In steps 512 and 514, the process is very similar to 410 and 412 of FIG.4, wherein the Outlier Index is generated for each user based on theOutlier Scores generated across the set of events, and whereinsub-populations are curated from the full Population of Participantsbased at least in part on the Outlier Index values generated for each ofthe participants in the full population. In some embodiments, othervalues are used as well, for example user meta-data regarding eachuser's self-assessment of knowledge in the subject at hand, eitherabsolute or relate to the average participant in the population. Theinventive process of curating a sub-population involves the inventivesteps of (a) removing participants from the population based at least inpart upon their Outlier Index being above a certain value, and/or (b)generating a weighting factor for participants within the population,the weighting factor based at least in part upon that user's OutlierIndex as compared to other users' Outlier Indices. Said weightingfactors are then used by subsequent crowd-based and/or swarm-basedmethods for generating insights from the group.

Referring again to FIGS. 4 and 5, both methods involve collecting a setof predictions from each member of the population of participants, theset of predictions reflecting a predicted outcome for each event in aset of events. In many embodiments, each prediction also includes one ormore confidence metrics. To make this concrete, the participants may begiven a survey that requires them to answer a set of questions eithertextually or verbally. For example, if the set of events was a series of5 baseball games, each of which has a binary outcome (i.e. one of thetwo teams will win the game), the set of questions given to theparticipants may be configured as shown in FIG. 6.

Each participant responds to the questions of FIG. 6, resulting in a setof predictions for each of the five games (i.e. the set of events),along with confidence metrics. Since this is a binary outcome event, themethod described in FIG. 5 may be used (although the method of FIG. 4works as well). Using the method of FIG. 5, the method determines thepercentage of participants who predicted the majority outcome for eachof the five games. In the rare case that the group is perfectly split,there is no majority outcome. To avoid this, the pool of participantscan be crafted to include an odd number of members.

Once the Majority Outcome Percentage has been computed for each of thefive events (games), the Majority Outcome Percentages are comparedagainst the Super-Majority Threshold to determine which events should beidentified as a Super Majority. In this example, let's say that three ofthe five games were supported by a Majority Outcome Percentage that ishigher than the Super-Majority Threshold.

For each of the three Super Majority events identified above, theprocess then assigns an Outlier Score to each participant. Thoseparticipants who chose the outcome that agrees with the Super Majorityoutcome are assigned a low Outlier score, while those participants whochose an outcome that goes against the Super Majority are assigned ahigh Outlier score. In the scoring method described in FIG. 5, the lowOutlier Score assigned to those who predicted the Super Majority outcomeis 0, and the high outlier score assigned to those who did not predictthe Super Majority outcome is 1. In some embodiments, the score isscaled down from 1 based upon the confidence metrics, wherein a lowconfidence metric decreases the score more than a high confidencemetric, as described above.

Finally, based on the Outlier Scores earned for each of the threeoutlier events, each participant is assigned an Outlier Index. If wetake the simple case where the outlier scores are either 0 or 1, theOutlier Index can be computed as a normalized value from 0 to 1, wherethe value is 0 if the user earned no outlier points, the value is 1/3(i.e. 0.33) if they earned 1 outlier point, is 2/3 if they earned 2outlier points, and the index is 3/3 (i.e. 1.0) if they earned 3 outlierpoints.

The final step is to curate a population from the full population basedon upon the outlier index values assigned to the participants. Asdescribed above, this can include (a) culling participants from the fullpopulations, and/or (b) weighting participants in subsequent crowd-basedand/or swarm-based prediction methods. For example, in a the very simplecase the process may be designed to remove participants who earned anoutlier index that is greater than 0.67. Thus, all participants who wereassigned an outlier index above that score, are removed from thepopulation and do not participant in a statistical crowd or a real-timeswarm-based system. In addition, all participants who receive an outlierindex of 0.67 or below are assigned a weighting factor based upon theiroutlier index, which reduces their contribution to the subsequentcrowd-based or swarm-based prediction methods. In a simple example, theweighting factor is=(1−Outlier_Index) assigned to that user.

For example, when computing the statistical average prediction within a“wisdom of crowds” method, the contribution of each user to thatstatistical average can be weighted by (1−Outlier_Index) for that user.Similarly, when enabling users to participate in a real-time swarm, theUser Intent values that are applied in real time can be scaled by aweighting factor of (1−Outlier_Index) for that user. In this way, usersthat are statistically most likely to provide incorrect insights areeither removed from the population and/or have reduced influence on theoutcome. What is significant about this method is that it does not useany historical data about the accuracy of participants in priorforecasting events. It enables a fresh pool of participants to becurated into a population that will give amplified accuracy in manycases.

In some embodiments of the present invention, a plurality of values aregenerated for each participant within the population of participantsthat reflect that participant's overall character across the set ofevents being predicted. The Outlier Index is one such multi-event valuethat characterizes each participant with respect to the otherparticipants within the population across a set of events beingpredicted. In addition, a Confidence Index is generated in someembodiments of the present invention as a normalized aggregation of theconfidence values provided in conjunction with each prediction withinthe set of predictions. For example, in the sample set of questionsshown in FIG. 5, each prediction includes a Confidence Question on ascale of 0% to 100% (questions 2, 6, 10, 14, and 18). For each user, theConfidence Index is the average confidence the user reports across thefull set of predictions, divided by the average confidence across allusers across all predictions in the set. This makes the Confidence Indexa normalized confidence value that can be compared across users. Inaddition, multi-event self-assessment values are also collected at theend of a session, after a participant has provided a full set ofpredictions. For example, as shown in FIG. 6, each participant was askedfour questions (21-24) after completing the set of predictions. Thequestions, shown here in generalized form, were:

-   -   (a) Predicted Self Accuracy: What Percentage of five game        outcomes above do you think you predicted correctly?    -   (b) Predicted Group Accuracy: What Percentage of the five game        outcomes above do you think the group, on average predicted        correctly?    -   (c) Self-Assessment of Knowledge: How knowledgeable do you        consider yourself with respect to the topic at hand?    -   (d) Group-Estimation of Knowledge: How knowledgeable do you        consider the average participant in the population?

Referring next to FIG. 7, a flowchart for a machine learning trainingphase process is shown in one embodiment of the present invention. Insome embodiments of the present invention, a plurality of multi-eventcharacterization values are computed during the data collection andanalysis process including (1) Outlier Index, (2) Confidence Index, (3)Predicted Self-Accuracy, (4) Predicted Group Accuracy, (5)Self-Assessment of Knowledge, and (6) Group-Estimation of Knowledge. Insuch embodiments, additional methods are added to the curation step(i.e. step 312 or step 414) wherein Machine Learning is used to find acorrelation between the multi-event characterization values and theperformance of participants when predicting events similar to the set ofevents.

In such embodiments (as shown in FIG. 7), a Training Phase is employedusing machine learning techniques such as regression analysis and/orclassification analysis employing one or more learning algorithms. Inthe first step 700, the training phrase is employed by first engaging alarge group of participants (for example 500 to 1000 participants). Inthe next step 702 the participants are employed to make predictionsacross a large set of events (for example, 20 to 30 baseball games). Aset of confidence metrics is also collected for each participant.

In the next step 704, for each of these 500 to 1000 participants, andacross the set of 20 to 30 events to be predicted, a set of values arecomputed including an Outlier Index (OI) and at least one or more of aConfidence Index (CI), a Predicted Self Accuracy (PSA), a PredictedGroup Accuracy (PGA), a Self-Assessment of Knowledge (SAK), and a GroupEstimation of Knowledge (GAK) (i.e. at least one value selected fromthat group).

In the next step 706, the events occur, and the event outcomes arecollected.

In step 708, user performance data is collected after the predictedevents have transpired (for example, after the 20 to 30 baseball gameshave been played). This data is then used to generate a score for eachof the large pool of participants, the score being an indication of howmany (or what percent) of the predicted events were forecast correctlyby each user. This value is generally computed as a normalized valuewith respect to the mean score earned across the large pool ofparticipants. This normalized value is referred to as a Normalized EventPrediction Score (NEPS).

The next step, 710, is the Training Phase wherein the machine learningsystem is trained (for example, using a regression analysis algorithm ora neural network system) to find a correlation between a plurality ofthe collected characterization values for a given user (i.e. at leasttwo from the group of the Outlier Index, the Confidence Index, aPredicted Self Accuracy, a Predicted Group Accuracy, a Self-Assessmentof Knowledge, and a Group Estimation of Knowledge) and the NormalizedEvent Prediction Score for a given user.

This correlation, once derived, can then be used in step 712 by theinventive methods herein on characterization value data collected fromnew users (new populations of users) to predict if the users are likelyto be a strong performer (i.e. have high normalized Event PredictionScores). In such embodiments, the machine learning system (for exampleusing multi-variant regression analysis) will provide a certainty metricas to whether or not a user with a particular combination ofcharacterization values (including an Outlier Index) is likely to be astrong or weak performer when making event predictions.

Thus, the final step 714 in the Optimization and Machine Learningprocess is to use the correlation that comes out of the Training Phaseof the machine learning system. Specifically, the trained model is usedby providing as input a set of characterization values for each memberof a new population of users, and generating as output a statisticalprofile for each member of the new population of users that predicts thelikelihood that each user will be a strong performer based only on theircharacterization values (not their historical performance). This is asignificant value because it enables a new population of participants tobe curated into a high performing sub-population even if historical datadoes not exist for those new participants.

Referring next to FIG. 8, a flowchart for a machine learning trainingphase process using characterization values is shown in one embodimentof the present invention.

In some embodiments of the present invention, the characterizationvalues for similarly performing users can be variable betweenpopulations and depending on the events responded to. In suchembodiments, additional methods are substituted for the Score GenerationSteps (i.e. step 704 and 710 of FIG. 7) where the characterizationvalues for each event a user responds to are aggregated into adescription of the distribution of those characterization values.

Steps 800 and 802 are similar to steps 700 and 702 of FIG. 7, where atraining population is engaged and predication sets and confidencemetrics sets are collected. In step 804, the characterization values foreach user's responses to each event are added into a CharacterizationDataset. In some embodiments, the Characterization Dataset takes theform of a software Dictionary object, with Participant Worker IDs asdictionary keys, and entries within the dictionary being an N×M matrix,where N is the number of events predicted, and M is the number ofCharacterization Value types for each event. Therefore, element (n,m)within the matrix would refer to game n, Characterization Value type m.

In step 806, For each characterization value type within thisCharacterization Dataset (e.g. Outcome Support Percentages or PredictedSelf Accuracy), a Probability Function is defined. This ProbabilityFunction acts as a metric to define the probability of the user's nextresponse to an arbitrary event for this type of Characterization Value.In some embodiments, this Probability Function takes the form of akernel-density estimate using Gaussian Kernels. In some embodiments,this takes the form of a Normalized Histogram with defined bin-widths.In some embodiments, this takes the form of a Normalized FourierTransform. In general, the Probability Function integrates to 1 over thespace in question, to form a proper measure of a probability densityfunction. For those embodiments where the Probability Function takes theform of a Kernel-Density estimate using Gaussian kernels, an exampleprobability function P(x) for a user i with n Characterization Valuesfor each response CV₁ . . . CV_(n) may be:

${P_{i}(x)} = {\frac{1}{n}{\sum\limits_{n}{\frac{1}{\sigma\sqrt{2\;\pi}}e^{- {0.5{\lbrack\frac{x - {CV}_{n}}{\sigma}\rbrack}}^{2}}}}}$

for some defined σ. In some embodiments σ is chosen throughexperimentation. In some embodiments σ is chosen through standard rulessuch as Silverman's Rule or Scott's Rule.

In the next step 808, once the Probability Function is defined for eachCharacterization Value type and is normalized to become a proper measureof probability across the Characterization Value space, MaximallyPredictive Locations are located for each Probability Function where theProbability Function at the Maximally Predictive Location is maximallypredictive of performance of users at that task. In some embodiments,the Maximally Predictive Locations take the form of singular locationswhere the Probability Function at the singular location is mostpredictive of performance of users at that task. In some embodiments,the Maximally Predictive Locations take the form of a number of singularlocations where the Probability Function at the singular locations ismost predictive of performance of users at that task. In someembodiments, the Maximally Predictive Locations are regions or areasthat are most predictive of performance of users. In some cases,predictiveness is measured using correlation between the ProbabilityFunction at the Maximally Predictive Location and the user'sperformance. In some cases, the number and location of MaximallyPredictive Location can be a function of the Probability Functionitself, such as finding all locations where the Probability Function isequal to a single value, or the locations where the Probability Functionhas local maxima. An example of a Probability Function that takes theform of a kernel-density estimate using Gaussian Kernels, applied tomany participants' Outcome Support Indexes, is shown below in FIG. 9 (asshown by the plurality of plotted participant probability functions904). A first maximally predictive location 900 is shown. A secondmaximally predictive location 902 is also shown. The maximallypredictive locations 900, 902 highlight two likely instances ofMaximally Predictive Locations for this data as an example of the typeof space that a Maximally Predictive Location falls in, from whichmultiple singular Maximally Predictive Locations may be selected forfurther analysis. There exists clear correlation between ProbabilityFunction value at these points and each agent's Normalized EventPrediction Score (plotted as line density in FIG. 9), indicating thatthe probability Function value may be a good predictor of agentperformance, as measured by the Normalized Event Prediction Score.

Once the Maximally Predictive Locations(s) are found, in step 810Probability Values may be calculated for each Probability Function atthe Maximally Predictive Location. In some cases, the Probability valuesare the value of the Probability Function at Maximally PredictiveLocation. In some cases, if the Maximally Predictive Location is afunction of the Probability Function, the Probability Values can be thedistances between Maximally Predictive Locations.

In step 812, the events occur, the event outcomes are collected and userperformance data is collected after the predicted events have transpiredand the NEPS for each user is generated as in the method of FIG. 7.

In step 814, a machine learning model is trained using the ProbabilityValues and other characteristic values for each participant as inputs,and the Normalized Event Predication Score for each user as outputvalue.

In the next step 816, a new population of participants is processed byhaving them predict a new set of events and provide confidence metrics.In step 818, once Probability Functions have been defined for eachCharacterization Value, and Maximally Predictive Location(s) have beenfound for each Probability Function, and a method for assigningProbability Values to users in a Training Population has been set, TheProbability Values for New Populations of Participants can be assignedusing the same Probability Functions, Maximally Predictive Location(s),and methods. These Probability Values are used to train Machine Learningsystems on Training Populations and curate New Populations based on thetrained Machine Learning systems.

Because these Probability Values describe the full distribution ofresponses, and do so using locations that are most related to theperformance of users on the events in question, the values bettercapture the spectrum of responses of variable, noisy, human-defineddata. The Probability Values have been shown experimentally to be morerobust to system noise such as population variability and humanvariability in question response, and extend well to multi-eventenvironments with variable number of events while predicting with higheraccuracy a given population's performances. Such Probability Valuesprovide significant inventive value because they enable a new populationof participants to be more accurately and more flexibly curated into ahigh performing sub-population than previously described methods, usingno historical data.

In the final step 820, the sub-population is curated based at least inpart on the statistical indicators for the participants in the newpopulation.

For illustrative purposes, an example embodiment of the method disclosedin FIG. 8 is provided as follows. A population of participants isengaged to predict whether the person depicted in each of a series of 20videos has generated their smile as a result of a genuine or fakeemotion. Data is collected from each of the participants using, forexample, a survey structure as disclosed previously. Once the data iscollected, the software system of the present invention may calculatethe Characterization Values desired, for example the Outcome SupportPercentages for each answer each user gave to the set of 20 videos.These Characterization Values are organized by user identifier in adictionary object comprising a Characterization Dataset, with eachuser's dictionary entry corresponding to a 1×20 matrix ofCharacterization Values, the first row of which represents the OutcomeSupport Percentage for each of the User's answers. For both types ofCharacterization value, we may assign a Probability Function that takesthe form of a kernel-density estimate using Gaussian Kernels.

The plot of this Probability Function as applied to each user's OutcomeSupport Percentages is shown in FIGS. 9 and 10. For clarity, the usersare divided into subgroups, where FIG. 9 shows the Probability functionfor users with high user performance (“Good Forecasters”) and FIG. 10shows the Probability Function for users with low user performance (“BadForecasters”). Each line 904, 1004 in FIGS. 9 and 10 represents a user,with the line optionally shaded by how well the user performed withinthe subset of users. Maximally Predictive Locations 900, 902, 1000, 1002are shown. We may select Maximally Predictive Locations 900, 902, 1000,1002 visually, then record the Probability Values from the probabilityfunction line 904 1004 for each user at the selected MaximallyPredictive Locations 900, 902, 1000, 1002. For this example, say weselect Maximally Predictive Locations 900, 902, 1000, 1002 of 30 and 63for Outcome Support Percentages. These two Probability Values, sampledfor each user, along with other singular Characterization Values such asPredicted Self Accuracy and Predicted Group Accuracy may then be used totrain a Machine Learning Algorithm that predicts each participant'sNormalized Event Prediction Score.

Once this Machine Learning Algorithm is trained, the system of thepresent invention can be used to process a new population of users,accepting and storing their responses to similar questions, calculatingtheir Characterization Values and Probability Values using the sameMaximally Predictive Locations and Probability Functions as before, andpredicting their Normalized Event Prediction Scores using the trainedMachine Learning Algorithm. For example, if we have 100 new participantsin the new population, and we want to predictively select asub-population of 25 optimized participants in order curate ahigh-performing Crowd or Swarm, the system of the present invention canbe configured to choose the 25 participants with the highest NormalizedEvent Prediction Scores. These 25 participants form a sub-population(denoted a Refined Population). Empirical testing has demonstratedthrough repeated trials that such a Refined Population, when used tomake predictions as a Crowd or Swarm, generally performs significantlybetter than a randomly-sampled 25 agents from the 100 new agentpopulation (denoted Random Population).

In the smile evaluation example above, we've shown through extensivetesting that an individual who is part of the Refined Population in thismethod gets on average 10.6% more questions correct in this dataset thanthe Random Population, or 1.163 times more questions correct.

In the realm of crowd-based predictions, if we consider the majorityvote of the 25 participants of each sampled population as a CrowdResponse, the Random Population's Crowd Response is correct 78.9% of thetime, while the Refined Population's Crowd Response is correct 85.5% ofthe time, a reduction in the number of incorrect answers of 31.1% overthe Random Population. Additionally, the standard deviation in number ofanswers correct of Crowd Responses for Random Crowds is 1.54, while thestandard deviation of that of Refined Crowds is only 0.62, a 60%reduction in Crowd response variability. Therefore, we are able tocreate more accurate and more stable crowd responses, which is importantif we want to run only 1 swarm and still have a reliable set ofpredictions. This is a highly significant result generated by using theinventive system disclosed herein. It's especially significant as itdoes not require any historical knowledge about the participantsregarding their past performance on similar tasks.

Similarly, in the realm of swarm-based predictions, when participatingwithin a real-time closed-loop swarm, such as disclosed in the relatedapplications, the Refined Population also performs better than the CrowdResponse accuracy quoted here, and still outperforms the RandomPopulation.

Clearly, there's a significant performance advantage to using themethods and systems disclosed herein to curate populations of humanparticipants, which comes from the inventive ability of being able topredict with higher accuracy which participants will perform well tocreating significantly higher-performing and more reliable crowds andswarms.

While many embodiments are described herein, it is appreciated that thisinvention can have a range of variations that practice the same basicmethods and achieve the novel collaborative capabilities that have beendisclosed above. Many of the functional units described in thisspecification have been labeled as modules, in order to moreparticularly emphasize their implementation independence. For example, amodule may be implemented as a hardware circuit comprising custom VLSIcircuits or gate arrays, off-the-shelf semiconductors such as logicchips, transistors, or other discrete components. A module may also beimplemented in programmable hardware devices such as field programmablegate arrays, programmable array logic, programmable logic devices or thelike.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions that may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code could be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork.

While the invention herein disclosed has been described by means ofspecific embodiments, examples and applications thereof, numerousmodifications and variations could be made thereto by those skilled inthe art without departing from the scope of the invention set forth inthe claims.

What is claimed is:
 1. A system for curating an optimized population ofhuman forecasting participants from a baseline population of humanforecasting participants based on an algorithmic analysis of predictiondata collected from each participant, the analysis identifying thelikelihood that each participant will be a high-performer in aprediction task involving one or more future events, the systemcomprising: a processing device including a processor and configured fornetwork communication; a plurality of application instances wherein eachapplication instance is configured to query a participant, receive inputfrom the queried participant about the prediction task, and be innetwork communication with the processing device regarding theprediction task, wherein the system is configured to perform the stepsof: query each member of the baseline population of participants aboutthe prediction task comprised of predicting a set of events, whereineach event has a set of possible outcomes including only two possibleoutcomes; collect a set of predictions from each participant, eachparticipant interacting with one application instance, wherein each setof predictions includes a predicted outcome for each event of the set ofevents; for each event in the set of events, compute one or more supportvalues wherein each support value for each event represents thepercentage of participants in the baseline population that predicted aparticular outcome within the set of possible outcomes; for eachparticipant, compute an outlier score for each event, wherein theoutlier score is computed by algorithmically corn paring theparticipant's predicted outcome for that event to the support value forthat outcome of that event, wherein the outlier score indicates how wellthat participant's prediction aligns with the predictions given by thebaseline population; for each participant, determine an outlier indexbased on the plurality of outlier scores computed for that participantfor the set of events, the outlier index indicating how well the set ofpredictions provided by that participant aligned with the sets ofpredictions given by the baseline population; curating an optimizedpopulation from the baseline population based at least in part upon aplurality of the outlier indexes, the curation process including atleast one selected from the group of (a) culling a plurality ofparticipants from the baseline population in response to the outlierindex of each culled participant indicating low alignment compared toother participants, and (b) generating a weighting value for a set ofparticipants in the baseline population, the generated weighting valuesbeing lower for participants with an outlier index indicating lowalignment as compared to weighting values for participants with anoutlier index indicating high alignment; using curated populationinformation to generate at least one crowd-based or swarm-basedprediction for a future event having at least two outcomes; andcomparing the larger support value for each event with a super-majoritythreshold percentage, wherein a super-majority indicator is assigned toeach event where the larger support value exceeds the super-majorityindicator.
 2. The system for curating the optimized population ofparticipants of claim 1, further comprising the system configured toperform the step of: inviting users to participate, wherein the baselinepopulation of participants comprises users who accept the invitation. 3.The system for curating the optimized population of participants ofclaim 1, further comprising the system configured to perform the stepof: collecting personal information from the baseline population ofparticipants.
 4. The system for curating the optimized population ofparticipants of claim 1, further comprising the system configured toperform the step of: after determining the outlier index, selecting asub-population of participants based at least in part on the outlierindex values.
 5. The system for curating the optimized population ofparticipants of claim 1, further comprising the system configured toperform the step of: collecting, from each participant, a quantitativeassessment of the participant's confidence in the participant'spredicted outcome for one event.
 6. The system for curating theoptimized population of participants of claim 5, the system furtherconfigured to perform the step of: weighting the support values based onat least one confidence assessment.
 7. The system for curating theoptimized population of participants of claim 1, further comprising thesystem configured to perform the step of: collecting, from eachparticipant, a quantitative assessment of the participant's confidencein their knowledge of a specific knowledge category.
 8. A method forcurating an optimized population of human forecasting participants froma baseline population of human forecasting participants based on analgorithmic analysis of prediction data collected from each participant,the analysis identifying the likelihood that each participant will be ahigh-performer in a prediction task involving one or more future events,comprising the steps of: querying, by a processing device including aprocessor and configured for networked communication, each member of thebaseline population of participants about the prediction task comprisedof predicting a set of events, wherein each event has a set of possibleoutcomes including only two possible outcomes; collecting, by aplurality of application instances, wherein each application instancereceives input from one participant and is in networked communicationwith the processing device, a set of predictions from each participant,each participant interacting with one application instance, wherein eachset of predictions includes a predicted outcome for each event of theset of events; computing, by the processor for each event in the set ofevents, one or more support values wherein each support value for eachevent represents the percentage of participants in the baselinepopulation that predicted a particular outcome within the set ofpossible outcomes; computing, by the processor for each participant, anoutlier score for each event, wherein the outlier score is computed byalgorithmically comparing the participant's predicted outcome for thatevent to the support value for that outcome of that event, wherein theoutlier score indicates how well that participant's prediction alignswith the predictions given by the baseline population; and determining,by the processor for each participant, an outlier index based on theplurality of outlier scores computed for that participant for the set ofevents, the outlier index indicating how well the set of predictionsprovided by that participant aligned with the sets of predictions givenby the baseline population; curating, by the processor, an optimizedpopulation from the baseline population based at least in part upon aplurality of the outlier indexes, the curation process including atleast one selected from the group of (a) culling a plurality ofparticipants from the baseline population in response to the outlierindex of each culled participant indicating low alignment compared toother participants, and (b) generating a weighting value for a set ofparticipants in the baseline population, the generated weighting valuesbeing lower for participants with an outlier index indicating lowalignment as compared to weighting values for participants with anoutlier index indicating high alignment; using, by the processor, ofcurated population information to generate at least one crowd-based orswarm-based prediction for a future event having at least two outcomes;and comparing the larger support value for each event with asuper-majority threshold percentage, wherein a super-majority indicatoris assigned to each event where the larger support value exceeds thesuper-majority indicator.
 9. The method for curating the optimizedpopulation of participants of claim 8, further comprising the step of:inviting users to participate, wherein the baseline population ofparticipants comprises users who accept the invitation.
 10. The methodfor curating the optimized population of participants of claim 8,further comprising the step of: collecting, by the applicationinstances, personal information from the baseline population ofparticipants.
 11. The method for curating the optimized population ofparticipants of claim 8, further comprising the step of: selecting, bythe processor after determining the outlier index, a sub-population ofparticipants based at least in part on the outlier index values.
 12. Themethod for curating the optimized population of participants of claim 8,further comprising the step of: collecting, by the application instancesfrom each participant, a quantitative assessment of the participant'sconfidence in the participant's predicted outcome for one event.
 13. Themethod for curating the optimized population of participants of claim12, further comprising the step of: weighting the support values basedon at least one confidence assessment.
 14. The method for curating theoptimized population of participants of claim 8, further comprising thestep of: collecting, by the application instances from each participant,a quantitative assessment of the participant's confidence in theirknowledge of a specific knowledge category.